Automatic speech recognition (ASR) systems use microphones and speakers to generate audible voice prompts to users and then receive speech in response to those voice prompts. The received speech can then be analyzed to determine its content. ASR systems generally deliver the audible voice prompts to the user as part of a conversational exchange between the ASR system and the user. However, designers of ASR systems also recognize that the systems may not always be able to appreciate when the generated audible prompts are not providing helpful information. To remedy this, the ASR systems can listen for speech from the user while the audible prompt is played. When the user speaks while the audible prompt is played, the ASR system can immediately end the prompt and begin analyzing the incoming speech. However, sometimes the ASR system detects sound other than the user's voice. When that occurs, the ASR system may interrupt prompts even though the user remained silent. The interruption of audible prompts caused by sounds other than the user's voice can cause frustration and reduce ASR performance and/or satisfaction.